Welcome back to deep learning and today I want to tell you a bit about attention mechanisms

and how they can be employed in order to build better deep neural networks.

Okay so let's focus your attention on attention and attention mechanisms.

So what is attention?

So you see that we humans process data actively by shifting our focus.

So we can focus on specific parts of images because they carry different information and

of course for some words for example we can derive the correct meaning only by means of

context and therefore you want to shift your attention and the way how you shift your attention

may also change the interpretation.

So you want to remember specific related events in the past in order to influence a certain

decision.

And this then allows to follow one thought at a time while suppressing information irrelevant

for the task.

So this is the idea you only want to focus on the relevant information.

One example that you could think of is the cocktail party problem where you have many

different people talking and you just focus on one person and by specific means like by

looking at the lips of that person you can focus your attention on the lips and then

you can also kind of use your stereo hearing as a kind of beam former to only listen into

that particular direction and then you are able to concentrate only on a single person,

the person that you are talking to using this kind of attention mechanism and we do that

quite successfully because otherwise we would be completely incapable of communication during

a cocktail party.

So what is the idea?

Well, you already seen those saliency maps and you could argue that we only want to look

at the pixel that are relevant for our decision in order to make the decision.

In fact we will start not with images but we want to talk first about sequence to sequence

models and here you can see a visualization of gradients from a CNN type of model that

is used to translate from English to German and if you now start plotting those gradients,

so this is essentially the visualization technique that we already looked at in image processing,

you can now see the gradient with respect to that particular input of the respective

output.

And if you do so you can notice that in most cases you see this essentially linear relationship

that English and German have of course very similar sequences in terms of words but then

you see that the actually beginning of the sequence which starts with to reach the official

residency of Prime Minister Nawaz Sharif is then translated to the offizielle residence

des Premier Ministers Nawaz Sharif zu erreichen.

So zu erreichen is to reach but to reach is first in English and it goes last in German.

So there is a long temporal context between the two but these two words essentially translate

to those two.

So we can use this kind of information that we can generate with these gradient back propagation

in order to figure out which parts of the input sequence are related to which parts

of the output sequence.

Now the question is how can we model this in order to boost the performance and let's

look at a typical translation model.

So what is typically done in sequence to sequence model if you work with recurrent neural networks

what you do is you have a forward pass in an encoder network and this receives some

input sequence and from the input sequence as we discussed earlier then we can compute

the hidden states h1 to ht.

So we have to process the entire sequence in order to compute ht and then ht is used

as a context vector for the decoder network so ht is essentially the representation of